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A method of producing an image of at least a 


tographic image, the lens, and the photographic receiv- 



portion of a photographic image onto a photographic re- er in response to pixels of the digital image and illumi- 
ceiver including receiving a digital image corresponding nating a portion of the photographic image of high sub- 
to the photographic image, the digital image comprising ject content to produce an image of such portion onto 
pixels, and locating the relative optical position of a pho- the photographic receiver. 
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Description 

[0001] The invention relates to producing an image of a portion of a photographic image onto a receiver by using 
digital image processing. 

5 [0002] For many decades, traditional commercial photofinishing systems have placed limits on the features offered 
to consumers to promote mass production. Among those features that are unavailable conventionally, zooming and 
cropping have been identified by both consumers and photofinishers as extremely useful additional features that could 
potentially improve the quality of the finished photographs and the subsequent picture sharing experiences. With the 
advent of, and rapid advances in digital imaging, many of the technical barriers that existed in traditional photography 
10 no longer stand insurmountable. 

[0003] Hybrid and digital photography provide the ability to crop undesirable content from a picture, and magnify or 
zoom the desired content to fill the entire photographic print. In spite of the fact that some traditional cameras with 
zoom capability provide consumers greater control over composing the desired scene content, studies have found that 
photographers may still wish to perform a certain amount of cropping and zooming when viewing the finished photo- 
's graph at a later time. Imprecise viewfinders of many point-and-shoot cameras, as well as simply second-guessing their 
initial compositions, are factors in the desirability of zoom and crop. In addition, it maybe desirable to use some other 
regular border templates such as ovals, heart shapes, squares, and so forth. In another scenario, some people com- 
monly referred to as "scrapbookers" tend to perform more aggressive crop in making a scrapbook, for example, cutting 
along the boundary of objects. 

20 [0004] There are significant differences in objectives and behaviors between these two types of cropping, namely 
album-making and scrapbook making, with the latter more difficult to understand and summarize. The invention de- 
scribed below performs automatic zooming and cropping for making photographic prints. One customer focus group 
study indicated that it would be beneficial to provide customers a double set of prints- one regular and one zoom. 
Moreover, it is preferred that the cropping and zooming be done automatically. Most customers do not want to think 

25 about how the zooming and cropping is being done as long as the content and quality (for example, sharpness) of the 
cropped and zoomed pictures is acceptable. 

[0005] There has been little research on automatic zoom and crop due to the apparent difficulty involved in performing 
such a task. None of the known conventional image manipulation software uses scene content in determining the 
automatic crop amount. For example, a program entitled "XV", a freeware package developed by John Bradley at 
30 University of Pennsylvania, USA (Department of Computer and Information Science), provides an "autocrop" function 
for manipulating images and operates in the following way: 

1 . The program examines a border line of an image, in all of the four directions, namely from the top, bottom, left 
and right sides; 

35 2. The program checks the variation within the line. In grayscale images, a line has to be uniform to be cropped. 

In color images, both the spatial correlation and spectral correlation have to be low, except for a small percentage 
of pixels, for the line to be qualified for cropping. In other words, a line will not be cropped if it contains a significant 
amount of variation; 

3. If a line along one dimension passes the criterion, the next line (row or column) inward is then examined; and 
40 4. The final cropped image is determined when the above recursive process stops. 

[0006] This program essentially tries to remove relatively homogeneous margins around the borders of an image. It 
does not examine the overall content of the image. In practice, the XV program is effective in cropping out the dark 
border generated due to imprecise alignment during the scanning process. However, disastrous results can often be 

45 produced due to the apparent lack of scene understanding. In some extreme cases, the entire image can be cropped. 
[0007] Another conventional system, described by Bollman and others, in US-A-5, 978,51 9 provides a method for 
cropping images based upon the different intensity levels within the image. With this system, an image to be cropped 
is scaled down to a grid and divided into non-overlapping blocks. The mean and variance of intensity levels are cal- 
culated for each block. Based on the distribution of variances in the blocks, a threshold is selected for the variance. 

so All blocks with a variance higher than the threshold variance are selected as regions of interest. The regions of interest 
are then cropped to a bounding rectangle. However, such a system is only effective when uncropped images contain 
regions where intensity levels are uniform and other regions where intensity levels vary considerably. The effectiveness 
of such a system is expected to be comparable to that of the XV program. The difference is that the XV program 
examines the image in a line by line fashion to identify uniform areas, while Bollman examines the image in a block 

55 by block fashion to identify uniform areas. In summary, both techniques cannot deal with images with non-uniform 
background. 

[0008] Some optical printing systems have the capability of changing the optical magnification of the relay lens used 
in the photographic copying process. In US-A-5, 995,201 , Sakaguchi describes a method of varying the effective mag- 
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nification of prints made from film originals utilizing a fixed optical lens instead of zoom lens. In US-A-5, 872,61 9, 
Stephenson and others, describe a method of printing photographs from a processed photographic filmstrip having 
images of different widths measured longitudinally of the filmstrip and having heights measured transversely of the 
filmstrip. This method uses a photographic printer having a zoom lens and a printing mask to provide printed images 

5 having a selected print width and a selected print height. In US-A-4,809,064 : Amos and others, describe an apparatus 
for printing a selected region of a photographic negative onto a photosensitive paper to form an enlarged and cropped 
photographic print. This apparatus includes means for projecting the photographic negative onto first and second zoom 
lenses, each of the zoom lenses having an adjustable magnification. In US-A-,872,643, Maeda and others, describe 
a film reproducing apparatus that can effectively perform zoom and crop. This apparatus includes an image pick-up 

10 device which picks up a film frame image recorded on a film to generate image data, an information reader which reads 
information about photographing conditions of the film frame image, and a reproducing area designator which desig- 
nates a reproducing area of the film frame image. 

However, the reproducing area of the film frame image is determined based on pre-recorded information about the 
position of the main object, as indicated by which zone of the photograph the automatic focusing (AF) operation in the 
is camera was on - part of the recorded information about photographing conditions. In all the above mentioned optical 
printing systems, the position of the photographic film sample and magnification factor of the relay lens are preselected. 
[0009] It is an object of the present invention to provide a method for producing a portion of an image from a photo- 
graphic image by identifying the main subject of the photographic image. 

[0010] This object is achieved by a method of producing an image of at least a portion of a photographic image onto 
20 a photographic receiver, comprising the steps of: 

a) receiving a digital image corresponding to the photographic image, the digital image comprising pixels; and 

b) locating the relative optical position of a photographic image, the lens, and the photographic receiver in response 
to pixels of the digital image and illuminating a portion of the photographic image of high subject content to produce 

25 an image of such portion onto the photographic receiver. 

ADVANTAGES 

[0011] One advantage of the invention lies in the ability to automatically crop and zoom photographic images based 
30 upon the scene contents. The digital image processing steps employed by the present invention includes a step of 
identifying the main subject within the digital image. The present invention uses the identified main subject of the digital 
image to automatically zoom and crop the image. Therefore, the present invention produces high-quality zoomed or 
cropped images automatically, regardless whether the background is uniform or not. 

[001 2] The foregoing and other objects, aspects and advantages will be better understood from the following detailed 
35 description of a preferred embodiment of the invention with reference to the drawings, in which: 

FIG. 1 is a schematic diagram of a system embodiment of the invention; 

FIG. 2 is a schematic architectural diagram of an embodiment of the invention; 

FIG. 3 is a schematic architectural diagram of an embodiment of the invention; 
40 FIG. 4 is a schematic architectural diagram of an embodiment of the invention; 

FIG. 5 is a schematic architectural diagram of an embodiment of the invention; 

FIG. 6 illustrates the application of the invention to a simulated photograph; 

FIG. 7 illustrates the application of the invention to a simulated photograph; 

FIG. 8 illustrates the application of the invention to a simulated photograph; 
45 FIG. 9 illustrates the application of the invention to a simulated photograph; 

FIG. 10 illustrates the application of the invention to a simulated photograph; 

FIG. 11 illustrates the application of the invention to a simulated photograph; 

FIG. 12 illustrates the application of the invention to a simulated photograph; 

FIG. 13 is an exemplary uncropped photograph; 
so FIG. 14 is a belief map of the image shown in FIG. 13; 

FIG. 15 is a cropped version of the image shown in FIG. 13; 

FIG. 1 7 is a belief map of the image shown in FIG. 1 6; and 

FIG. 18 is a cropped version of the image shown in FIG. 16. 

55 [0013] The invention automatically zooms and crops digital images according to an analysis of the main subject in 
the scene. Previously, a system for detecting main subjects (for example, main subject detection or "MSD") in a con- 
sumer-type photographic image from the perspective of a third-party observer has been developed. Main subject de- 
tection provides a measure of saliency or relative importance for different regions that are associated with different 
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subjects in an image. Main subject detection enables a discriminative treatment of the scene content for a number of 
applications related to consumer photographic images, including automatic crop and zoom. 

[0014] Conventional wisdom in the field of computer vision, which reflects how a human observer would perform 
such tasks as main subject detection and cropping, calls for a problem-solving path via object recognition and scene 
content determination according to the semantic meaning of recognized objects. However, generic object recognition 
remains a largely unsolved problem despite decades of effort from academia and industry. 

[0015] The MSD system is built upon mostly low-level vision features with semantic information integrated whenever 
available. This MSD system has a number of sub-tasks, including region segmentation, perceptual grouping, feature 
extraction, and probabilistic and semantic reasoning. In particular, a large number of features are extracted for each 
segmented region in the image to represent a wide variety of visual saliency properties, which are then input into a 
tunable, extensible probability network to generate a belief map containing a continuum of values. 
[0016] Using MSD, regions that belong to the main subject are generally differentiated from the background clutter 
in the image. Thus, automatic zoom and crop becomes possible. Automatic zoom and crop is a nontrivial operation 
that was considered impossible for unconstrained images, which do not necessarily contain uniform background, with- 
out a certain amount of scene understanding. In the absence of content-driven cropping, conventional systems have 
concentrated on simply using a centered crop at a fixed zoom (magnification) factor, or removing the uniform back- 
ground touching the image borders. The centered crop has been found unappealing to customers. 
[0017] The output of MSD used by the invention is a list of segmented regions ranked in descending order of their 
likelihood (or belief) as potential main subjects for a generic or specific application. This list can be readily converted 
into a map in which the brightness of a region is proportional to the main subject belief of the region. Therefore, this 
map can be called a main subject "belief map. This "belief" map is more than a binary map that only indicates location 
of the determined main subject. The associated likelihood is also attached to each region so that regions with large 
values correspond to regions with high confidence or belief of being part of the main subject. 

[0018] To some extent, this belief map reflects the inherent uncertainty for humans to perform such a task as MSD 
because different observers may disagree on certain subject matter while agreeing on other subject matter in terms 
of main subjects. However, a binary decision, when desired, can be readily obtained by using an appropriate threshold 
on the belief map. Moreover, the belief information may be very useful for downstream applications. For example, 
different weighting factors can be assigned to different regions (subject matters) in determining the amount of crop. 
[0019] For determination of crop, the invention uses the main subject belief map instead of a binarized version of 
30 the map to avoid making a bad cropping decision that is irreversible. Furthermore, using the continuous values of the 
main subject beliefs helps trade-off different regions under the constraints encountered in cropping. A binary decision 
on what to include and what not to include, once made ; leaves little room for trade-off. For example, if the main subject 
region is smaller than the crop window, the only reasonable choice, given a binary main subject map, is to leave equal 
amounts of margin around the main subject region . On the other hand, secondary main subjects are indicated by lower 
belief values in the main subject belief map, and can be included according to a descending order of belief values once 
the main subject of highest belief values are included. Moreover, if an undesirable binary decision on what to include/ 
exclude is made, there is no recourse to correct the mistake. Consequently, the cropping result becomes sensitive to 
the threshold used to obtain the binary decision. With a continuous-valued main subject belief map, every region or 
object is associated with a likelihood of being included or a belief value in its being included. 

[0020] To reduce the degrees of freedom in determining the amount of crop, in particular for making photographic 
prints, in one embodiment, the invention restricts the set of allowable zoom factors to {1 .5x, 2x, 3x, 4x}. This is based 
on the findings in the customer focus studies. Those skilled in the art would recognize that the present invention could 
be used with any the zoom factor. In one example, the default zoom factor is set at 1 .5X. 

[0021] "Digital image understanding technology" as used herein means technology which is specifically designed to 
exclusively perform on a digital image representing at least a portion of an original scene captured by an image capture 
device other than a copier or scanner, the functions of: 

(i) digitally processing such digital image (other than by color and/or luminance, that is, non-spatial analysis of 
color) to recognize and thereby assign useful meaning to: 

50 

a human understandable object other than (a) a human-created graphic object, including without limitation, a 
diagram, chart, alphanumeric character, mark, symbol, pattern and any combination of the foregoing, (b) the 
entire original scene, or (c) any part of such original scene completely designated by a human in such original 
scene; 

55 an object attribute (including without limitation, characteristics, movement or direction, of such human under- 

standable object) in such original scene (other than for using such object attribute to control the functions, 
including image signal processing functions, of the image capture device); or 

a scene condition (other than for using such scene condition to control the functions, including image signal 



35 



40 



45 



BNSDCCID: <EP 1 158464A1_I_> 



EP1 158 464 A1 

processing functions, of the image capture device) of such original scene; and 



5 



(ii) based on the results of step (i) and in accordance with such useful meaning, modifying, transforming, com- 
pressing, expanding, enhancing, editing, classifying, formatting, transmitting, or storing the digital image repre- 
senting at least a portion of the original scene, orcausing an operation other than image processing to be performed. 



General Description of an Optical Printer System Application 

[0022] Referring to FIG. 1 , the following description relates to an optical printing system. A photographic film sample 
10 31 is received by a film scanner 32 which produces a source digital image 1 0 relating to the spatial density distribution 
of the photographic film sample. This source digital image is received by a digital image processor 20. The digital image 
processor 20 may be connected to a general control computer 40 under operator control from an input control device 
60. The monitor device 50 displays diagnostic information about the optical printing system. The general control com- 
puter 40 keeps track of the lens magnification setting. 
15 [0023] Referring to FIG. 2, a zoom factor 1 1 , which corresponds to the lens magnification setting may also received 
by the image processor 20 from the general control computer 40 under operator control. The image processor 20 
receives the source digital image 10 and uses the zoom factor 11 and the source digital image 10 to calculate the 
proper position for the photographic film sample in the form of a film sample position 9. The photographic film sample 
is positioned in a gate device 36 which holds the film negative in place during the exposure. The gate device 36 receives 
20 the film sample position 9 to position the photographic film sample to adjust which portion of the imaging area of the 
photograph will be printed. 

[0024] Referring to FIG. 1 , a lamp house 34 provides the illumination source which is transmitted through the pho- 
tographic film sample 31 and focused by a lens 12 onto photographic paper 38. The time integration device 13 opens 
and closes a shutter for a variable length of time allowing the focused light from the lamp house 34 to expose the 
25 photographic paper 38. The exposure control device 16 receives a brightness balance value from the digital image 
processor 20. The exposure control device 16 uses the brightness balance value to regulate the length of time the 
shutter of the time integration device stays open. 

[0025] A block diagram of the inventive cropping process (for example, the digital image understanding technology) 
is shown in FIG. 3, which is discussed in relation to FIGS. 6-12. FIGS. 6-12 illustrate the inventive process being applied 

30 to an original image shown in FIG. 6. 

[0026] In item 200, the belief map is created using MSD. The invention selects a zoom factor (for example 1 .5X) and 
a crop window 80 (as shown in FIG. 7), as referred to in item 201 of FIG. 3. This zoom factor may be selected by an 
operator, or by an automatic method based directly on the main subject belief map (for example, an estimate of the 
size of the main subject). The crop window is typically a rectangular window with a certain aspect ratio. If the zoom 

35 factor is selected by the operator, the value of the zoom factor is sent from the general control computer 40 to the 
digital image processor 20 shown in FIG. 1 . For both the operator selected case or the main subject belief map case, 
the zoom factor is used to communicate with the lens 1 2 to adjust the lens magnification setting. This adjustment allows 
the lens 12 to image the appropriate size of the photographic film sample 31 onto the photographic paper 38. 
[0027] In item 202, regions of the belief map are clustered and the lowest belief cluster (for example, the background 

40 belief) is set to zero using a predefined threshold. As discussed in greater detail below, sections of the image having 
a belief value below a certain threshold are considered background sections. In item 202 such sections are given a 
belief of zero for purposes of this embodiment of the invention. 

[0028] Then, in item 203 the centroid, or center-of-mass, of nonzero beliefs are computed and, as shown in item 
204, the crop window position is centered at the centroid, as shown in FIG. 7. More specifically, in FIG. 6 the subject 
45 having the highest belief in the belief map is the woman and the stroller, FIG. 7 illustrates that the centroid of this 
subject is approximately the top of the baby's head. 

[0029] The centroid (x,y) of a belief map is calculated using the following procedure: 



where x t and y, denote that coordinates of a pixel in the belief map and 
bel(Xf,y f ) represents the belief value at this pixel location. 
55 [0030] The crop window is then moved so that the entire crop window is within the original image (for example item 
205) as shown in FIG. 8. In item 207, the crop window 80 is moved again so that all the regions of the highest belief 
values ("main subject") are included within the crop window and to create a margin 81 , as shown in FIG. 9. This process 
(for example, 207) captures the entire subject of interest. Therefore, as shown in FIG. 9, the top of the woman's head 
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is included in the crop window. Compare this to FIG . 8 where the top of the woman's head was outside the crop window. 
[0031] Decision box 208 determines whether an acceptable solution has been found, that is, whether it is possible 
to include at least the regions of the highest belief values in the crop window. 

[0032] If an acceptable solution exists, the window is again moved, as shown in item 210, to optimize a subject 
5 content index for the crop window. The preferred embodiment of the present invention defines the subject content 
index as the sum of belief values within the crop window. It should be noted that the present invention specifies higher 
numerical belief values corresponding to higher main subject probability. Therefore, finding a numerical maximum of 
the sum of the belief values is equivalent to finding an optimum of the subject content index. This is shown in FIG. 10 
where the secondary objects (for example flowers) are included within the crop window 80 to increase the sum of 
10 beliefs. The sum of beliefs for a crop window is computed as follows. 



15 



25 



(jt,y)ew 



where bel(x,y) represents the belief value at a given pixel location (x,y) within the crop window w. 
[0033] Provided that the primary subjects are included, moving the crop window so that more of the secondary 
subjects are included would increase the sum of belief values within the crop window. Recall that the primary subjects 
20 are indicated by the highest belief values and the secondary subjects are indicated by belief values lower than those 
of the primary subjects but higher than those of the background subjects. The goal is to find the crop window that has 
the highest sum of belief values while ensuring that the primary subjects are completely included in the crop window, 
that is, 



where W denotes the set of all possible crop windows that satisfy all the aforementioned constraints (for example, 

30 those that are completely within the uncropped image and those that encompass the entire primary subjects). 

[0034] Then, in item 212, the position of the center of the crop window is used to calculate thetranslational component 
of the film sample position 9. The gate device 36, shown in FIG. 1 , receives the film sample position 9 and uses this 
information to control the position of the photographic film sample 31 relative to the lens 12. Those skilled in the art 
will recognize that either or both of the lens 12 and the photographic film sample 31 may be moved to achieve the 

35 centering of the effective cropped image region on the photographic paper 38. 

[0035] Referring to FIG. 3, if decision box 208 does not produce an acceptable solution, the digital image is rotated 
by 90 degrees, as shown in item 206, and the processing returns to item 205. At the second and greater pass through 
decision box 209, if the crop window 80 has already been rotated by 90 degrees and there is still not an acceptable 
solution (from decision box 208), then the sum of the beliefs for the centroid-based crop window 80 are computed for 

^0 both orientations (for example, original and 90 degrees), as shown in item 211 . Then, the position of the center of the 
crop window is used to calculate the translational component of the film sample position 9 and the rotational component 
of the film sample position 9 is set to 90 degrees. The gate device 36, shown in FIG. 1 , receives the film sample position 
9 and uses this information to control the position of the photographic film sample 31 relative to the lens 12. The gate 
device 36 then rotates the photographic film sample 31 by 90 degrees. 

45 [0036] As would be known by one ordinarily skilled in the art given this disclosure, the photographic film sample may 
be rotated at an arbitrary angles (for example 45 o ,180*, and so forth.) in item 206. Further, the decision box 209 can 
allow multiple passes of the process described in items 205, 207 at different rotation angles in an attempt to find the 
orientation having the highest sum of beliefs. 

[0037] The simulated image example shown in FIGS. 6-12 illustrates the progress the invention makes as it moves 
so through the process shown in FIG. 3. One could formulate the problem as a global exhaustive search for the best 
solution. The procedure used in the invention is considered a "greedy" searching approach and is certainly more efficient 
than conventional processes. 

[0038] FIGS. 4 and 5 illustrate variations of the process shown in FIG. 3 of determining the film sample position 9. 
FIG. 4 compares a landscape crop with a portrait crop, regardless of the initial (not necessarily the correct) picture 
55 orientation. The second variation in FIG. 5 creates a series of "ring-around" cropped versions with a zoom factor series 
(for example, {1x : 1 .5x, 2x, 3x, and 4x}) for presentation to a human operator. 

[0039] More specifically, in FIG. 4 items 300-303 are substantially identical to items 200-203 in FIG. 3. In item 304, 
the crop window 80 is positioned centered atthecentroid without any rotation. Simultaneously, in a parallel processing 
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step, the position of the crop window 80 is centered at the centroid with a rotation (for example, 90 degree rotation). 
In items 306 and 307 the optimal crop window 80 is determined, as discussed above with respect to FIG. 3. This 
produces cropped images 308 and 309. In item 31 0 the better (for example, higher sum of beliefs) of the two cropped 
images is selected and the corresponding translational and rotational components of the film sample position 9 are 
5 recorded. 

[0040] Referring to FIG. 5, items 400-407 are substantially identical to items 300-307 discussed above. However, in 
the process shown in FIG. 5, decision box 41 0 observes the cropped images 408, 409 produced by the parallel process- 
ing in items 404-407 and determines whether an acceptable level has been attained. If the processing in item 410 
determines that a different zoom factor or a different crop window 80 should be evaluated, processing returns to item 
10 401 . This process is repeated until an acceptable result is produced, ending in item 411 . In similar fashion, the corre- 
sponding translational and rotational components of the film sample position 9 are recorded. 

[0041] The invention utilizes a built-in M k-means" clustering process to determine proper thresholds of MSD beliefs 
for each application. The invention also uses clustering, as discussed below to enhance the cropping process. In one 
preferred embodiment, it is sufficient to use three levels to quantize MSD beliefs, namely "high", "medium", and "low." 

15 As would be known by one ordinarily skilled in the art, the invention is not limited to simply three levels of classification, 
but instead can utilize a reasonable number of classification levels to reduce the (unnecessary) variation in the belief 
map. These three levels allow for the main subject (high), the background (low), and an intermediate level (medium) 
to capture secondary subjects, or uncertainty, or salient regions of background. Therefore, the invention can perform 
a k-means clustering with k = 3 on the MSD belief map to "quantize" the beliefs. Consequently, the belief for each 

20 region is replaced by the mean belief of the cluster in that region. Note that a k-means clustering with k = 2 essentially 
produces a binary map with two clusters, "high" and "low," which is undesirable for cropping based on earlier discussion. 
[0042] There are two major advantages in performing such clustering or quantization. First, clustering helps back- 
ground separation by grouping low-belief background regions togetherto form a uniformly low-belief (for example, zero 
belief) background region. Second, clustering helps remove noise in belief ordering by grouping similar belief levels 

25 together. The centroiding operation does not need such quantization (nor should it be affected by the quantization). 
The main purpose of the quantization used here is to provide a threshold for the background. 

[0043] The k-means clustering effectively performs a multi-level thresholding operation to the belief map. After clus- 
tering, two thresholds can be determined as follows: 

30 

thresholds = (C low+ C medium ) /2, threshold high = {C medium + C high )/2 

where {C toMr C medium , C high ] is the set of centroids (average belief values) for the three clusters, and threshold fow and 
threshold high are the low and high thresholds, respectively. 

35 [0044] Regions with belief values below the lower threshold are considered "background" and their belief values are 
set to zero in items 202, 302 and 402 discussed above. Regions with belief values above the higher threshold are 
considered part of the main subject and need to be included in their entirety, whenever possible. Regions with inter- 
mediate belief values (for example, less than or equal to the higher threshold and greater than or equal to the lower 
threshold) are considered part of the "secondary subject' and will be included as a whole or partially, if possible, to 

40 maximize the sum of main subject belief values retained by the crop window. Note that the variance statistics on the 
three clusters can be used to set the thresholds more accurately to reflect cluster dispersions. 

[0045] The invention initializes the k-means process by finding the maximum value bel maximum and minimum values 
bet mmimum of tne belief map, computing the average value bel average of the maximum and minimum values for item in 
the belief map, and setting the initial centroids (denoted by a superscript of 0) at these three values, that is, 

45 

^low — bel mlnimium , C mec ff um = bel medjum , C high = bel maximum 

Other ways of initialization may apply. For more about the k-means process, see Sonka, Hfavac, and Boyle, Image 
so Procesing Analysis, and Machine Vision, PWS Publishing, 1999 pagse 307-308. For typical MSD belief maps, the k- 
means process usually converges in fewer than 10 iterations. 

[0046] In applications where a zoom version of the cropped area is desired, there are two scenarios to consider. 
First, the zoom version effectively requires higher spatial resolution than the highest resolution of the original data. 
However, a visible loss of image sharpness is likely of concern in the situation. Second, the zoom version effectively 
55 requires lower spatial resolution than the highest resolution of the original data. In both cases, the invention uses an 
interpolation process to resample the data in order to retain a maximum amount of image detail. In general, edge or 
detail-preserving image interpolation processes such as cubic-spline interpolation are preferred because they tend to 
preserve the detail and sharpness of the original image better. 
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[0047] Example consumer photographs and their various cropped versions are shown in pictures "house" (for ex- 
ample, FIGS. 13-15) and "volleyball" (FIGS. 16-18). More specifically, FIGS. 13 and 16 illustrate uncropped original 
photographic images. FIGS. 14 and 17 illustrate belief maps, with lighter regions indicating higher belief values. As 
would be known by one ordinarily skilled in the art given this disclosure, the light intensity variations shown in FIGS. 
14 and 17 are readily converted into numerical values for calculating the sum of the belief values discussed above. 
Finally, FIGS. 15 and 18 illustrate images cropped according to the invention. 

[0048] For the "house" picture, both Bradley and Boliman (US-A-5,978,51 9) would keep the entire image and not be 
able to produce a cropped image because of the shadows at the bottom and the tree extending to the top border of 
the uncropped image (FIG. 13). There are no continuous flat background regions extending from the image borders 
in this picture, as required by US-A-5,978,51 9. Similarly the top of the tree in FIG. 16 would not be cropped in the 
system disclosed in US-A-5,978,51 9. 

[0049] Secondary subjects can lead to a more balanced cropped picture. For the "volleyball" picture (FIG. 1 6), the 
inclusion of some parts of the tree by the algorithm leads to more interesting cropped pictures than simply placing the 
main subjects (players) in the center of the cropped image (FIG. 18). The invention was able to do so because the 
trees are indicated to be of secondary importance based on the belief map FIG. 17. It is obvious that the art taught by 
Bradley and Boliman in US-A-5,978,51 9 would not be able to produce such a nicely cropped image. In fact, both 
Bradley and Boliman (US-A-5,978,51 9) would at best remove the entire lower lawn portion of the picture and keep the 
tree branches in the upper-left of the uncropped image. 

[0050] A computer program product may include one or more storage medium, for example; magnetic storage media 
such as magnetic disk (such as a floppy disk) or magnetic tape; optical storage media such as optical disk, optical 
tape, or machine readable bar code; solid-state electronic storage devices such as random access memory (RAM), 
or read-only memory (ROM); or any other physical device or media employed to store a computer program having 
instructions for practicing a method according to the present invention. 
[0051] Other features of the invention are included below. 

[0052] The method further comprising repeating said positioning procedure with a second zoom factor and a second 
crop window and determining if said second zoom factor and said second crop window produces a higher subject 
content index. ~ J 

[0053] The method wherein said zoom factor is selected according to a size of said main subject cluster 
[0054] The method further comprising positioning said crop window such that the subject content index of said crop 
window is at an optimum. 

[0055] The method further comprising positioning said crop window such that said crop window includes all of said 
main subject cluster. 

[0056] The method further comprising positioning said crop window to include a buffer around said main subject 
cluster. 

[0057] A computer storage product having at least one computer storage medium having instructions stored therein. 
PARTS LIST 



[0058] 
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film sample position 


10 


source digital image 


11 


zoom factor 


12 


lens 


13 


time integration device 


20 


digital image processor 


31 


photographic film sample 


32 


film scanner 


34 


lamp house 


36 


gate device 


38 


photographic paper 


40 


general control computer 


50 


monitor device 


60 


input control device 


80 


crop window 


81 


margin 


200 


image and belief map 


201 


decision box for zoom factor selection 
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202 decision box for clustering regions 

203 decision box for computing centroids 

204 decision box for positioning the crop window 

205 decision box moving a window 

5 206 decision box for rotating the image 

207 decision box for moving a window with the highest belief 

208 decision box for determining if a solution exists 

209 decision box for determining if the image is rotated 

210 decision box for computing the sum of beliefs 
10 21 1 decision box for computing the sum of beliefs 

300 image and belief map 

301 decision box for selecting a zoom factor and crop window 

302 decision box for setting the background belief to 0 

303 decision box for computing the centroid of nonzero beliefs 
is 304 decision box for positioning the crop window 

305 decision box for positioning the crop window 

306 decision box for determining the optimal crop 

307 decision box for determining the optimal crop 

308 cropped image # 1 
20 309 cropped image U 2 

310 decision box for selecting the better crop 

31 1 cropped image 

400 image and belief map 

401 decision box for selecting a zoom factor and crop window 
?s 402 decision box for setting the background belief to 0 

403 decision box for computing the centroid of nonzero beliefs 

404 decision box for positioning the crop window 

405 decision box for positioning the crop window 

406 decision box for determining the optimal crop 
30 407 decision box for determining the optimal crop 

408 cropped image # n 

409 cropped image # n+1 

410 decision box for selecting a different zoom factor or different crop window 

41 1 decision box for ending the process 

35 

Claims 

1 . A method of producing an image of at least a portion of a photographic image onto a photographic receiver, com- 
40 prising the steps of: 

a) receiving a digital image corresponding to the photographic image, the digital image comprising pixels; and 

b) locating the relative optical position of a photographic image, the lens, and the photographic receiver in 
response to pixels of the digital image and illuminating a portion of the photographic image of high subject 

45 content to produce an image of such portion onto the photographic receiver. 

2. A method of producing an image of at least a portion of a photographic image onto a photographic receiver, com- 
prising the steps of: 

so a) receiving a digital image corresponding to the photographic image, the digital image comprising pixels; 

b) developing a belief map of the digital image, by using pixels of the digital image to determine a series of 
features, and using such features to assign the probability of the location of a main subject of the digital image 
in the belief map; and 

c) locating the relative optical position of a photographic image, the lens, and the photographic receiver in 
55 response to the belief map and illuminating a portion of the photographic image of high subject content to 

produce an image of such portion onto the photographic receiver. 

3. A method of producing an image of a portion of at least a portion of a photographic image onto a photographic 
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receiver, comprising the steps of: 

a) receiving a digital image corresponding to the photographic image, the digital image comprising pixels; 

b) developing a belief map of the digital image, by using pixels of the digital image to determine a series of 
features, and using such features to assign the probability of the location of a main subject of the digital image 
in the belief map; 

c) selecting a crop window of a size and shape; 

d) selecting an initial position of the crop window; 

e) using the belief values corresponding to the crop window to select the position of the crop window; and 

f) locating the relative optical position of a photographic image, the lens, and the photographic receiver in 
response to the position of the crop window within the digital image and illuminating a portion of the photo- 
graphic image to produce an image onto the photographic receiver. 

4. A method as set forth in claim 3 wherein step c) further comprises the steps of: 

i) receiving a magnification factor; and 

ii) using the magnification factor to select the size of the crop window. 

5. A method as set forth in claim 3 wherein step e) further comprising the steps of: 

i) calculating a subject content index for the crop window derived from the belief values; 

ii) following a positioning procedure of repeating step i) for at least two positions of the crop window; and 

iii) use the subject content index values to select the crop window position. 

^5 6. A method as set forth in claim 5 wherein step d) further comprises the steps of: 

i) using the belief map to determine the center of the mass of the belief map; and 

ii) using the center of mass of the belief map to select the initial position of the crop window. 

30 7. The method of claim 6 wherein the crop window is completely within the digital image. 

8. The method of claim 3 wherein step b) further comprises the step of performing a clustering of the belief map to 
identify at least a cluster of highest belief values corresponding to main subject, a cluster of intermediate belief 
values corresponding to secondary subjects, and a cluster of lowest belief values corresponding to the background. 



15 



20 



35 



9. The method of claim 8 wherein said clustering includes setting said background portions to a zero belief value. 



10. The method of claim 5 further comprising repeating said positioning procedure with a rotated image and determining 
if said rotated image produces a higher subject content index. 

40 



45 



50 



55 
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